Expect: Programmed Dialogue
***************************

This appendix presents the same information available in the Expect reference-manual entry (``man page''); we include it in the DejaGnu manual for convenient reference.  The author of Expect, and of this reference, is Don Libes, of the National Institute of Standards and Technology (NIST).

Introduction
============

Expect is a program that ``talks'' to other interactive programs according to a script.  Following the script, Expect knows what can be expected from a program and what the correct response should be.  An interpreted language provides branching and high-level control structures to direct the dialogue.  In addition, the user can take control and interact directly when desired, afterward returning control to the script.

expectk is a mixture of Expect and Tk.  It behaves just like Expect and Tk's wish.  Expect can also be used directly in C or C++ (that is, without Tcl). See libexpect(3).

The name ``Expect'' comes from the idea of ``send/expect'' sequences popularized by uucp, kermit, and other modem control programs.  However, unlike uucp, Expect is generalized so that it can be run as a user-level command with any program and task in mind.  (Expect can actually talk to several programs at the same time.)

For example, here are some things Expect can do:

  * Cause your computer to dial you back, so that you can login without paying for the call.

  * Start a game (e.g., rogue) and if the optimal configuration doesn't appear, restart it (again and again) until it does, then hand over control to you.

  * Run fsck, and in response to its questions, answer ``yes'' or ``no'', or give control back to you, based on predetermined criteria.

  * Connect to another network or BBS (e.g., MCI Mail, CompuServe) and automatically retrieve your mail so that it appears as if it was originally sent to your local system.

  * Carry environment variables, current directory, or any kind of information across rlogin, telnet, tip, su, chgrp, and so on.

There are a variety of reasons why the shell cannot perform these tasks. (Try, you'll see.)  All are possible with Expect.

In general, Expect is useful for running any program which requires interaction between the program and the user.  All that is necessary is that the interaction can be characterized programmatically. Expect can also give the user back control (without halting the program being controlled) if desired.  Similarly, the user can return control to the script at any time.

Usage
=====

Expect reads a file for a list of commands to execute.  Expect may also be invoked implicitly on systems which support the #! notation by marking the script executable, and making the first line in your script:

     #!/usr/local/bin/expect -f

Of course, the path must accurately describe where Expect lives. /usr/local/bin is just an example.

Here is a summary of the command line options you can use with Expect:

     expect  [ -dinN ]  [ -c CMDS ]  [[  -f ] CMDFILE ]  [ ARGS ]

-- may be used to delimit the end of the options.  This is useful if you want to pass an option-like argument to your script without it being interpreted by Expect.  This can usefully be placed in the #! line to prevent any option-like interpretation by Expect.  For example, the following will leave the original arguments (including the script name) in the variable ARGV.

     #!/usr/local/bin/expect --

Note that the usual getopt and execve conventions must be observed when adding arguments to the #! line.

Expect understands these command line options:

-c CMDS
     Execute the command CMDS before any in the script.  The command should be quoted to prevent being broken up by the shell.  This option may be used multiple times.  Multiple commands may be executed with a single -c by separating them with semicolons.  Commands are executed in the order they appear.

-d
     Enables some debugging output, which primarily reports internal activity of commands such as expect and interact.  This option has the same effect as debug 1 at the beginning of an Expect script, plus the version of Expect is printed.  (The strace command is useful for tracing statements, and the trace command is useful for tracing variable assignments.)

-f CMDFILE
     Read commands from CMDFILE.  The option itself is optional as it is only useful with using the #! notation in a script(see above), so that other arguments may be supplied on the command line.

     If the string - is supplied as a filename, standard input is read instead.  (Use ./- to read from a file actually named -.)

-i
     Causes Expect to interactively prompt for commands instead of reading them from a file.  Prompting is terminated via the exit command or upon EOF.  See interpreter (below) for more information. -i is assumed if neither a command file nor -c is used.

-N
-n
     The file $expect_library/expect.rc is sourced automatically if present, unless the -N option is used.  Immediately after this, the file ~/.expect.rc is sourced automatically, unless the -n option is used.  Both of these are sourced after executing any -c options.

ARGS
     Optional ARGS are constructed into a list and stored in the variable named argv.  argv may be accessed with the usual list access commands.  For example, the length of argv is calculated by llength $argv.

     argv[0] is defined to be the name of the script.  For example, the following prints out the name of the script and the first two arguments:

          send_user [lrange $argv 0 2]

Commands
========

Expect uses Tcl (Tool Command Language).  Tcl provides control flow (e.g., if, for, break), expression evaluation and several other features such as recursion, procedure definition, etc.  Commands used here but not defined (e.g., set, if, exec) are Tcl commands (see Tcl).  Expect supports additional commands, described below.  Unless otherwise specified, commands return the empty string.

Commands are listed alphabetically so that they can be quickly located. However, new users may find it easier to start by reading the descriptions of spawn, send, expect, and interact, in that order.  Then read the examples at the rear of this man page.  (In the text of this man page, Expect with an uppercase "E" refers to the Expect program while expect with a lower-case "e" refers to the expect command within the Expect program.)

close [ -i SPAWN_ID ]
     Closes the connection to the current process.  Most interactive programs will detect EOF on their stdin and exit; thus close usually suffices to kill the process as well.  The -i option declares the process to close corresponding to the named SPAWN_ID.

     Both expect and interact will detect when the current process exits and implicitly do a close.  But if you kill the process by, say, exec kill $pid, you will need to explicitly call close.

     No matter whether the connection is closed implicitly or explicitly, you should call wait to clear up the corresponding kernel process slot.  close does not call wait since there is no guarantee that closing a process connection will cause it to exit.  See wait below for more info.

debug [ -f FILE ] VALUE
     Causes further commands to send debugging information internal to Expect to stderr if VALUE is non-zero.  This output is disabled if VALUE is 0.  The debugging information includes every character received, and every attempt made to match the current output against the patterns.

     If the optional FILE is supplied, all normal and debugging output is written to that file (regardless of the value of VALUE).  Any previous debugging output file is closed.

disconnect
     Disconnects a forked process from the terminal.  It continues running in the background.  The process is given its own process group (if possible).  Standard I/O is redirected to /dev/null.

     The following fragment uses disconnect to continue running the script in the background.

          if [fork]!=0 exit
          disconnect
          ...

     The following script reads a password, and then runs a program every hour that demands a password each time it is run.  The script supplies the password so that you only have to type it once.  (See the system command which demonstrates how to turn off password echoing.)

          send_user "password?\ "
          expect_user -re "(.*)\n"
          for {} 1 {} {
                if [fork]!=0 {exec sleep 3600;continue}
                disconnect
                spawn priv_prog
                expect Password:
                send "$expect_out(1,string)\r"
                ...
                exit
          }

     An advantage to using disconnect over the shell asynchronous process feature (&) is that Expect can save the terminal parameters prior to disconnection, and then later apply them to new ptys.  With &, Expect does not have a chance to read the terminal's parameters since the terminal is already disconnected by the time Expect receives control.

exit [ STATUS ]
     Kills Expect.  All connections to spawned processes are closed. Closure will be detected as an EOF by spawned processes.

     Exit generates a signal 0 (see trap), but otherwise takes no other actions beyond what the normal _exit procedure does.  Thus, spawned processes that do not check for EOF may continue to run.  (A variety of conditions are important to determining, for example, what signals a spawned process will be sent, but these are system-dependent.) Spawned processes that continue to run will be inherited by init.

     STATUS (or 0 if not specified) is returned as the exit status of Expect.  exit is implicitly executed if the end of the script is reached.

expect [[ -OPTS ] PAT1 BODY1 ] ... [ -OPTS ] PATN [ BODYN ]
     waits until one of the patterns matches the output of a spawned process, a specified time period has passed, or an end-of-file is seen.  If the final body is null, it may be omitted.

     Patterns from the most recent expect_before command are implicitly used before any other patterns.  Patterns from the most recent expect_after command are implicitly used after any other patterns.

     If the arguments to the entire expect statement require more than one line, all the arguments may be ``braced'' into one so as to avoid terminating each line with a backslash.  In this one case, the usual Tcl substitutions will occur despite the braces.

     If a pattern is the keyword eof, the corresponding body is executed upon end-of-file.  If a pattern is the keyword timeout, the corresponding body is executed upon timeout.  The default timeout period is 10 seconds but may be set, for example to 30, by the command set timeout 30.  An infinite timeout may be designated by the value -1.  If a pattern is the keyword default, the corresponding body is executed upon either timeout or end-of-file.

     If a pattern matches, then the corresponding body is executed. expect returns the result of the body (or null if no pattern matched). In the event that multiple patterns match, the one appearing first is used to select a body.

     Each time new output arrives, it is compared to each pattern in the order they are listed.  Thus, you may test for absence of a match by making the last pattern something guaranteed to appear, such as a prompt. In situations where there is no prompt, you must use timeout (just like you would if you were interacting manually).

     Patterns are specified in two ways.  By default, patterns are specified as with Tcl's string match command.  (Such patterns are also similar to C-shell filename expansion, usually referred to as ``glob'' patterns).

     For example, the following fragment looks for a successful login.  (Note that abort is presumed to be a procedure defined elsewhere in the script.)

          expect {
            connected            break
            busy                 {
                                   print busy\n
                                   continue
                                 }
            failed               abort
            "invalid password"   abort
            timeout              abort
          }

     Quotes are necessary on the fourth pattern since it contains a space, which would otherwise separate the pattern from the action.  Patterns with the same action (such as the 3rd and 4th) require listing the actions again.  This can be avoid by using regexp-style patterns (see below).  More information on forming glob-style patterns can be found in the Tcl manual.

     Alternatively, regexp-style patterns follow the syntax defined by Tcl's regexp (short for ``regular expression'') command.  regexp patterns are introduced with the option -re.  The previous example can be rewritten using a regexp as:

          expect {
            connected                     break
            busy                          {
                                            print busy\n
                                            continue
                                          }
            -re "failed|invalid password" abort
            timeout                       abort
          }

     Both types of patterns are ``unanchored''.  This means that patterns do not have to match the entire string, but can begin and end the match anywhere in the string (as long as everything else matches).  Use ^ to match the beginning of a string, and $ to match the end.  Note that if you do not wait for the end of a string, your responses can easily end up in the middle of the string as they are echoed from the spawned process.  While still producing correct results, the output can look unnatural.  Thus, use of $ is encouraged if you can exactly describe the characters at the end of a string.

     The -nocase option causes uppercase characters of the output to compare as if they were lowercase characters.  The pattern is not affected.

     While reading output, more than 2000 bytes can force earlier bytes to be ``forgotten''.  This may be changed with the function match_max. (Note that excessively large values can slow down the pattern matcher.) If PATLIST is full_buffer, the corresponding body is executed if MATCH_MAX bytes have been received and no other patterns have matched.

     Upon matching a pattern (or eof or buffer_full), any matching and previously unmatched output is saved in the array element expect_out(buffer).  Up to 9 regexp substring matches are saved in the array elements expect_out(1,string) through expect_out(9,string).  For each substring, expect_out(X,start) holds the starting index; the ending index is stored in expect_out(X,end).  These indices are in a form suitable for lrange; X corresponds to the substring position in the pattern.  0 refers to the entire pattern itself.  For example, if a process has produced output of abcdefgh\n, the result of:

          expect  "cd"

     is as if the following statements had executed:

          set expect_out(0,start) 2
          set expect_out(0,end) 3
          set expect_out(0,string) cd
          set expect_out(buffer) abcd

     and efgh\n is left in the output buffer. For the output abbbcabkkkka\n, the result of:

          expect -re "b(b*).*(k+)"

     is as if the following statements had executed:

          set expect_out(0,start) 1
          set expect_out(0,end) 10
          set expect_out(0,string) bbbcabkkkk
          set expect_out(1,start) 2
          set expect_out(1,end) 3
          set expect_out(1,string) bb
          set expect_out(2,start) 10
          set expect_out(2,end) 10
          set expect_out(2,string) k
          set expect_out(buffer) abbbcabkkkk

     and a\n is left in the output buffer.  The pattern * will flush the output buffer without reading any more output from the process.

     Normally, the matched output is discarded from Expect's internal buffers.  This may be prevented by prefixing a pattern with the -n option.  The name, placement, and existence of this option is subject to change in a future release.  Therefore, it should not be used in permanent scripts.  However, it is especially useful in experimenting (which is why it has a one-character name).

     By default, patterns are matched against output from the current process; however, the -i option declares the output from the named SPAWN_ID be matched against any following patterns (up to the next -i).  For example, the following example waits for connected from the current process, or busy, failed or invalid password from the SPAWN_ID named by $proc2.

          expect {
            connected                     break
            -i $proc2 busy                {
                                            print busy\n
                                            continue
                                          }
            -re "failed|invalid password" abort
            timeout                       abort
          }

     The variable any_spawn_id may be used to match patterns to any SPAWN_ID that is named with another -i option associated with a pattern.  Upon matching a pattern (or eof or buffer_full), the variable expect_out(spawn_id) is set to the SPAWN_ID which produced the matching output.

     Actions such as break and continue cause control structures (i.e., for, proc) to behave in the usual way. The special argument -expect to continue allows expect itself to continue executing rather than returning as it normally would.

     This is useful for avoiding explicit loops or repeated expect statements.  The following example is part of a fragment to automate rlogin.  The continue avoids having to write a second expect statement (to look for the prompt again) if the rlogin prompts for a password.

          expect {
            Password: {
              system stty -echo
              send_user "password (for $user) on $host: "
              expect_user -re "(.*)\n"
              send_user "\n"
              send "$expect_out(1,string)\r"
              system stty echo
              continue -expect
            } incorrect {
              send_user "invalid password or account\n"
              exit
            } timeout {
              send_user "connection to $host timed out\n"
              exit
            } eof {
              send_user "connection to $host failed: \
          $expect_out(buffer)"
              exit
            } -re $prompt
          }

     For example, the following fragment might help a user guide an interaction that is already totally automated.  In this case, the terminal is put into raw mode.  If the user presses +, a variable is incremented.  If p is pressed, several returns are sent to the process, perhaps to poke it in some way, and i lets the user interact with the process, effectively stealing away control from the script.  In each case, the continue -expect allows the current expect to continue pattern matching after executing the current action.

          system stty raw -echo
          expect_after { -i $user_spawn_id
            "p" {send "\r\r\r"; continue -expect}
            "+" {incr foo; continue -expect}
            "i" {interact; continue -expect}
            "quit" exit
          }

     continue -expect resets the timeout timer.

expect_after [ EXPECT ARGS ]
     Takes the same arguments as expect; however, it returns immediately.  Pattern-action pairs from the most recent expect_after are implicitly added to any following expect commands.  If a pattern matches, it is treated as if it had been specified in the expect command itself, and the associated body is executed in the context of the expect command.  If patterns from both expect and expect_after can match, the expect pattern is used.

     Unless overridden by a -i option, expect_after patterns match against the spawn_id defined at the time that the expect_after command was executed (not when its pattern is matched).

expect_before [ EXPECT ARGS ]
     takes the same arguments as expect; however, it returns immediately.  Pattern-action pairs from the most recent expect_before are implicitly added to any following expect commands.  If a pattern matches, it is treated as if it had been specified in the expect command itself, and the associated body is executed in the context of the expect command.  If patterns from both expect_before and expect can match, the expect_before pattern is used.

     Unless overridden by a -i option, expect_before patterns match against the spawn_id defined at the time that the expect_before command was executed (not when its pattern is matched).

expect_user [ EXPECT ARGS ]
     Like expect, but reads characters from stdin (i.e. keystrokes from the user).  By default, reading is performed in cooked mode.  Thus, lines must end with a return in order for expect to see them.  This may be changed via stty (see the system command below).

expect_version [[ -exit ] VERSION ]
     is useful for assuring that the script is compatible with the current version of Expect.

     With no arguments, the current version of Expect is returned. This version may then be encoded in your script.  If you actually know that you are not using features of recent versions, you can specify an earlier version.

     Versions consist of up to three numbers separated by dots.  First is the major number.  Scripts written for versions of Expect with a different major number will almost certainly not work. expect_version returns an error if the major numbers do not match.

     Second is the minor number.  Scripts written for a version with a greater minor number than the current version may depend upon some new feature and might not run.  expect_version returns an error if the major numbers match, but the script minor number is greater than that of the running Expect.

     Third is a number that plays no part in the version comparison. However, it is incremented when the Expect software distribution is changed in any way, such as by additional documentation or optimization.  It is reset to 0 upon each new minor version.

     With the -exit option, Expect prints an error and exits if the version is out of date.

     There have been three major versions of Expect.  The first was never officially released and only existed for two months, as I experimented and designed the basic style of Expect.  The second version lasted a year and a half until the time when Tcl 6 and Expect 3 were issued. Version 6 of Tcl was incompatible with earlier versions, but John Ousterhout (Tcl's author) suggested that enough experience had been gained that such changes were appropriate, and this might be the last time it could be done because further delay would be that much more painful due to the ever-growing number of people using it.  I feel the same way.  I hope that the current version of Expect will last many years without the introduction of incompatibilities that might render scripts obsolete.

     During its one and a half year public lifetime, the second version of Expect was requested (and perhaps even used) by over 3000 sites.  I received numerous suggestions for improvements or future directions. Many of these either appear in the current version or are addressed in the Expect FAQ file.

fork
     Creates a new process.  The new process is an exact copy of the current Expect process.  On success, fork returns 0 to the new (child) process and return the process ID of the child process to the parent process.  On failure (invariably due to lack of resources, e.g., swap space, memory), fork returns -1 to the parent process, and no child process is created.

     Forked processes exit via the exit command, just like the original process.  Forked processes are allowed to write to the log files.  If you do not disable debugging or logging in most of the processes, the result can be confusing.

     Some pty implementations may be confused by multiple readers and writers, even momentarily.  Thus, it is safest to fork before spawning processes.

getpid
     Returns the process id of the current process.

interact [ STRING1 BODY1 ] ... [ STRINGN [ BODYN ]]
     Gives control of the current process to the user, so that keystrokes are sent to the current process, and the stdout and stderr of the current process are returned.

     String-body pairs may be specified as arguments, in which case the body is executed when the corresponding string is entered.  (By default, the string is not sent to the current process.)  The interpreter command is assumed, if the final body is missing.

     If the arguments to the entire interact statement require more than one line, all the arguments may be ``braced'' into one so as to avoid terminating each line with a backslash.  In this one case, the usual Tcl substitutions will occur despite the braces.

     For example, the following command runs interact with the following string-body pairs defined: When C-Z is pressed, Expect is suspended.  When C-A is pressed, the user sees you typed a control-A and the process is sent a ^A.  When $ is pressed, the user sees the date.  When C-C is pressed, Expect exits.  If foo is entered, the user sees bar.  When ~~ is pressed, the Expect interpreter runs interactively.

          set CTRLZ \032
          interact {
            $CTRLZ  {exec kill -STOP 0}
            \001    {send_user "you typed a control-A\n";
                      send "\001"
                    }
            $       {send_user "The date is [exec date]."}
            \003    exit
            foo     {send_user "bar"}
            ~~
          }

     In string-body pairs, strings are matched in the order they are listed as arguments.  Strings that partially match are not sent to the current process in anticipation of the remainder coming.  If characters are then entered such that there can no longer possibly be a match, only the part of the string will be sent to the process that cannot possibly begin another match.  Thus, strings that are substrings of partial matches can match later, if the original strings that was attempting to be match ultimately fails.

     By default, string matching is exact with no wild cards.  (In contrast, the expect command uses glob-style patterns by default.)  The -re option forces the string to be interpreted as a regexp-style pattern.  In this case, matching substrings are stored in the variable interact_out similarly to the way expect stores its output in the variable expect_out.

     The option -eof introduces an action that is executed upon end-of-file.  The -eof option applies to the most recently specified process (such as via -input or -output).  If the -eof option precedes all spawned processes, then it applies to all spawned processes that do not have an -eof option.  The default -eof action is return, so that interact simply returns upon any EOF.

     The option -timeout introduces a timeout (in seconds) and action that is executed after no characters have been read for a given time. The -timeout option applies to the most recently specified process.  If the -timeout option precedes all spawned processes, then it applies to all spawned processes that do not have a -timeout option.  There is no default -timeout.  The special variable timeout (used by the expect command) has no affect on this timeout.

     For example, the following statement could be used to autologout users who have not typed anything for an hour but who still get frequent system messages:

          interact -input $user_spawn_id \
                   -output $spawn_id -f  \
                   -timeout 3600 return

     Actions such as break and continue cause control structures (i.e., for, proc) to behave in the usual way. However return causes interact to return to its caller, while return -tcl causes interact to cause a return in its caller.  For example, if proc foo called interact which then executed the action return -tcl, proc foo would return.  (This means that if interact calls interpreter interactively typing return will cause the interact to continue, while return -tcl will cause the interact to return to its caller.)

     During interact, raw mode is used so that all characters may be passed to the current process.  If the current process does not catch job control signals, it will stop if sent a stop signal (by default C-Z).  To restart it, send a continue signal (such as by kill -CONT PID).  If you really want to send a SIGSTOP to such a process (by C-Z), consider spawning csh first and then running your program.  On the other hand, if you want to send a SIGSTOP to Expect itself, first press ESC (the escape character), and then press C-Z.

     String-body pairs can be used as a shorthand for avoiding having to enter the interpreter and execute commands interactively.  The previous terminal mode is used while the body of a string-body pair is being executed.

     The -f option (for fast) skips the possibility of a temporary mode switch during pair processing.  This consequently prevents characters from being lost when the terminal is returned to raw mode (an unfortunate feature of the terminal driver) at the end of a key-body pair execution.  The only reason not to use -f is if your action depends on running in cooked mode.

     The -F option indicates that all following options behave as if they each were declared with -f.

     The previous example is restated below in a more efficient form, using -F.  The first line cannot use -f because it would leave the user back in the shell in raw mode.  The last line would remain in raw mode but it is overridden by the interpreter command itself, which forces it into cooked mode temporarily.  The second line requires no extra \r because send automatically adds one.  The other lines need no change and run fine with -f.

          set CTRLZ \032
          interact {
            $CTRLZ   {kill -STOP 0}
            -F \001  {send_user "you typed a control-A\n";
                       send "\001"
                     }
            $        {send_user "The date is [exec date]."}
            \003     exit
            foo      {send_user "bar"}
            ~~
          }

     By default, actions that change the value of spawn_id will not affect the behavior of interact even if input or output sources were originally associated with spawn_id.

     The -update option forces spawn_id to be reexamined after evaluation of an action.  This could be used, for example, so that pressing a particular function key would switch to interacting with a different process.

     The -echo option sends characters that match the following pattern back to the process that generated them as each character is read.  This may be useful when the user needs to see feedback from partially typed patterns.

     If a pattern is being echoed but eventually fails to match, the characters are sent to the spawned process.  If the spawned process then echoes them, the user will see the characters twice.  -echo is probably only appropriate in situations where the user is unlikely to not complete the pattern.  For example, the following excerpt is from rftp, the recursive-ftp script, where the user is prompted to enter ~g, ~p, or ~l, to get, put, or list the current directory recursively.  These are so far away from the normal ftp commands, that the user is unlikely to type ~ followed by anything else, except mistakenly, in which case, they'll probably just ignore the result anyway.

          interact {
            -echo ~g {getcurdirectory 1}
            -echo ~l {getcurdirectory 0}
            -echo ~p {putcurdirectory}
          }

     The -flush option sends characters that match the following pattern on to the output process as characters are read.

     This is useful when you wish to let a program echo back the pattern. For example, the following might be used to monitor where a person is dialing (a Hayes-style modem).  Each time atd is seen the script logs the rest of the line.

          proc lognumber {} {
            interact -flush -f -re "(.*)\r" return
            puts $log "[exec date]: dialed $interact_out(1,string)"
          }

          interact -flush -f "atd" lognumber

     During interact, previous use of log_user is ignored.  In particular, interact will force its output to be logged (sent to the standard output) since it is presumed the user doesn't wish to interact blindly.

     The -o option causes following key-body pairs to be applied to the output of the current process.  This can be useful, for example, when dealing with hosts that send unwanted characters during a telnet session.

     By default, interact expects the user to be writing stdin and reading stdout of the Expect process itself.  The -u option (for ``user'') makes interact look for the user as the process named by its argument (which must be a spawned id).

     This allows two unrelated processes to be joined together without using an explicit loop.  To aid in debugging, Expect diagnostics always go to stderr (or stdout for certain logging and debugging information).  For the same reason, the interpreter command will read interactively from stdin.

     For example, the following fragment creates a login process.  Then it dials the user (not shown), and finally connects the two together.  Of course, any process may be substituted for login.  A shell, for example, would allow the user to work without supplying an account and password.

          spawn login
          set login $spawn_id
          spawn tip modem
          ... ;# dial back out to user
          ... ;# connect user to login
          interact -u $login

     To send output to multiple processes, list each spawn id prefaced by a -output option.  Input for a group of output spawn ids may be determined by a spawn id prefaced by a -input option.  All following options and strings (or patterns) apply to this input until another -input option appears.  If no -input appears, -output implies -input $user_spawn_id -output. (Similarly, with patterns that do not have -input.)  If one -input is specified, it overrides $user_spawn_id.  If a second -input is specified, it overrides $spawn_id. Additional -input options may be specified.

     The two implied input processes default to having their outputs specified as $spawn_id and $user_spawn_id (in reverse). If a -input option appears with no -output option, characters from that process are discarded.

     The -i option introduces a replacement for the current spawn_id when no other -input or -output options are used.

interpreter
     causes the user to be interactively prompted for Expect and Tcl commands.  The result of each command is printed.

     Actions such as break and continue cause control structures (i.e., for, proc) to behave in the usual way. However return causes interpreter to return to its caller, while return -tcl causes interpreter to cause a return in its caller.  For example, if proc foo called interpreter which then executed the action .BR return -tcl , proc foo would return.  Any other command causes interpreter to continue prompting for new commands.

     By default, the prompt contains two integers.  The first integer describes the depth of the evaluation stack (i.e., how many procedures have yet to return).  The second integer is the Tcl history identifier. The prompt can be set by defining a procedure called prompt1 whose return value becomes the next prompt.  If a statement has open quotes, parens, braces, or brackets, a secondary prompt (by default +> ) is issued upon newline.  The secondary prompt may be set by defining a procedure called prompt2.

     During interpreter, cooked mode is used, even if the its caller was using raw mode.

log_file [[-a] FILE ]
     If a filename is provided, log_file will record a transcript of the session (beginning at that point) in the file.  log_file will stop recording if no argument is given.  Any previous log file is closed.

     The -a option forces output to be logged that was suppressed by the log_user command.

     The log_file command appends to old files rather than truncating them, for the convenience of being able to turn logging off and on multiple times in one session.  A simple way to always start with a fresh log file is to delete the log file before using the log_file command for the first time in a script.  For example:

          exec rm transcript
          log_file transcript

log_user EXPRESSION
     By default, the send/expect dialogue is logged to standard output (and a logfile if open).  This logging is disabled by the command log_user 0 and reenabled by log_user 1.

match_max [-d] [-i SPAWN_ID ] [SIZE]
     defines the size of the buffer (in bytes) used internally by expect.  With no SIZE argument, the current size is returned.

     With the -d option, the default size is set.  (The initial default is 2000.)  With the -i option, the size is set for the named spawn id, otherwise it is set for the current process.

overlay [-N SPAWN_ID...] PROGRAM [ARGS]
     Executes PROGRAM ARGS in place of the current Expect program, which terminates.  A bare hyphen argument forces a hyphen in front of the command name as if it was a login shell.  All spawn ids are closed except for those named as arguments.  These are mapped onto the named file descriptors N.

     Spawn ids are mapped to file descriptors for the new program to inherit. For example, the following line runs chess and allows it to be controlled by the current process---say, a chess master.

          overlay -0 $spawn_id -1 $spawn_id -2 $spawn_id chess

     This is more efficient than interact -u.  However, it sacrifices the ability to do programmed interaction since the Expect process is no longer in control.

     Note that no controlling terminal is provided.  Thus, if you disconnect or remap standard input, programs that do job control (shells, login, etc) will not function properly.

parity [-d] [-i SPAWN_ID] [VALUE]
     Defines whether parity should be retained or stripped from the output of spawned processes.  If VALUE is zero, parity is stripped, otherwise it is not stripped.  With no VALUE argument, the current value is returned.

     With the -d option, the default parity value is set.  (The initial default is 1, i.e., parity is not stripped.)  With the -i option, the parity value is set for the named spawn id, otherwise it is set for the current process.

send [-s] [-h] [-i SPAWN_ID] [-raw] ARGS
     Sends ARGS to the current process.  Strings are interpreted following Tcl rules.  For example, the command

          send "hello world\r"

     sends the characters h e l l o w o r l d RET to the current process.  (Tcl includes a command similar to printf (called format) which can build arbitrarily complex strings.)

     Characters are sent immediately although programs with line-buffered input will not read the characters until a return character is sent.  A return character is denoted \r.

     The -i option declares that the string be sent to the named spawn id.  If the spawn id is user_spawn_id, and the terminal is in raw mode, newlines in the string are translated to return-newline sequences so that they appear as it the terminal was in cooked mode.  The -raw option disables this translation.

     The -s option forces output to be sent ``slowly'', thus avoid the common situation where a computer outtypes an input buffer that was designed for a human who would never outtype the same buffer.  This output is controlled by the value of the variable send_slow which takes a two element list.  The first element is an integer that describes the number of bytes to send atomically.  The second element is a real number that describes the number of seconds by which the atomic sends must be separated.  For example, set send_slow 10 .001 would force send -s to send strings with 1 millisecond in between each 10 characters sent.

     The -h option forces output to be sent (somewhat) like a human actually typing.  Human-like delays appear between the characters.  (The algorithm is based upon a Weibull distribution, with modifications to suit this particular application.)  This output is controlled by the value of the variable send_human which takes a five element list. The first two elements are average interarrival time of characters in seconds.  The first is used by default.  The second is used at word endings, to simulate the subtle pauses that occasionally occur at such transitions.  The third parameter is a measure of variability, where .1 is quite variable, 1 is reasonably variable, and 10 is quite invariable. The extremes are 0 to infinity.  The last two parameters are, respectively, a minimum and maximum interarrival time.  As an example, the following command types a lot like the author (a fast and consistent typist):

          set send_human {.1 .3 1 .05 2}
          send -h "I'm hungry.  Let's do lunch."

     while the following might be more suitable after a hangover:

          set send_human {.4 .4 .2 .5 100}
          send -h "Goodd party lash night!"

     Note that errors are not simulated, although you can set up error correction situations yourself by embedding mistakes and corrections in a send argument.

     It is a good idea to precede the first send to a process by an expect.  expect will wait for the process to start, while send cannot.  In particular, if the first send completes before the process starts running, you run the risk of having your data ignored.  In situations where interactive programs offer no initial prompt, you can precede send by a delay as in:

          # To avoid hints on how to break in,
          # there is no prompt for an external password.
          # Wait for 5 seconds for exec to complete
          spawn telnet very.secure.gov
          exec sleep 5
          send password\r

send_error ARGS
     Like send, except that the arguments are sent to standard error rather than the current process.

send_log ARGS
     Like send, except that the arguments are only sent to the log file (see log_file).  The arguments are ignored if no log file is open.

send_spawn ARGS
     An alias for send.  If you are use expectk or some other variant of Expect in the Tk environment, send is defined by Tk for an entirely different purpose.  send_spawn is provided for compatibility between environments.

send_user ARGS
     is like send, except that the arguments are sent to stdout rather than the current process.

spawn [ARGS] program [ARGS]
     Creates a new process running PROGRAM ARGS.  Its stdin, stdout, and stderr are connected to Expect, so that they may be read and written by other Expect commands.  The connection is broken by close or if the process itself closes any of the file descriptors.

     When a process is started by spawn, the variable spawn_id is set to a descriptor referring to that process.  The process described by spawn_id is considered the current process. spawn_id may be read or written, in effect providing job control.

     user_spawn_id is a predefined variable containing a descriptor which refers to the user.  For example, when spawn_id is set to this value, expect behaves like expect_user.  Do not assume the value of user_spawn_id will remain the same from one version of Expect to another.

     tty_spawn_id is a predefined variable containing a descriptor for /dev/tty.  This may be the same as user_spawn_id but may be different if user_spawn_id has been redirected.  If /dev/tty does not exist (such as in a cron, at, or batch script), then tty_spawn_id is not defined.  This may be tested as:

          if [info vars tty_spawn_id] {
            # /dev/tty exists
            ...
          } else {
            # /dev/tty doesn't exist
            # probably in cron, batch, or at script
            ...
          }

     spawn returns the UNIX process id.  Note that the UNIX process id is not equivalent to the descriptor in spawn_id.

     By default, spawn echoes the command name and arguments.  The -noecho argument stops spawn from doing this.

     The option -console causes console output to be redirected to the spawned process.  This is not supported on all systems.

     Internally, spawn uses a pty, initialized the same way as the user's tty.  This is further initialized so that all settings are sane (according to stty(1)).  If the variable stty_init is defined, it is interpreted in the style of stty arguments as further configuration.  For example, set stty_init raw will cause further spawned processes's terminals to start in raw mode.  -nottycopy skips the initialization based on the user's tty.  -nottyinit skips the sane initialization.

     Normally, spawn takes little time to execute.  If you notice spawn taking a significant amount of time, it is probably encountering ptys that are wedged.  A number of tests are run on ptys to avoid entanglements with errant processes.  (These take 10 seconds per wedged pty.)  Running Expect with the -d option will show if Expect is encountering many ptys in odd states.  If you cannot kill the processes to which these ptys are attached, your only recourse may be to reboot.

     If PROGRAM cannot be spawned successfully because exec fails (e.g. when PROGRAM does not exist), an error message will be returned by the next interact or expect command as if PROGRAM had run and produced the error message as output.  This behavior is a natural consequence of the implementation of spawn. Internally, spawn forks, after which the spawned process has no way to communicate with the original Expect process except by communication via the spawn id.

strace LEVEL
     Causes following statements to be printed before being executed.  (The Tcl trace command traces variables.)  LEVEL indicates how far down in the call stack to trace.  For example, the following command runs Expect while tracing the first 4 levels of calls, but none below that.

          expect -c "strace 4" script.exp

system ARGS
     Gives ARGS to sh as input, just if it had been typed as a command from a terminal.  Expect waits until the shell terminates.  The return status from sh is handled the same way that exec handles its return status.

     In contrast to exec which redirects stdin and stdout to the script, system performs no redirection (other than that indicated by the string itself).  Thus, it is possible to use programs which must talk directly to /dev/tty.  For the same reason, the results of system are not recorded in the log.

     system understands and evaluates certain cases of stty directly, in order to efficiently handle mode switching during interpeter and interact.  In particular, the arguments raw or -cooked put the terminal into raw mode.  The arguments -raw or cooked put the terminal into cooked mode.  The arguments echo and -echo put the terminal into echo and noecho mode respectively.

     The following example illustrates how to use system to temporarily disable echoing.  This could be used in otherwise-automatic scripts to avoid embedding passwords in them.  (For more discussion of this, see Expect Hints.)

          system stty -echo
          send_user "Password: "
          expect_user -re "(.*)\n"
          set password $expect_out(1,string)
          system stty echo

trap [[COMMAND] SIGNALS]
     Causes the given COMMAND to be executed upon future receipt of any of the given signals.  If COMMAND is absent, the signal actions are reset to their defaults.  If COMMAND is the string SIG_IGN, the signals are ignored.  SIGNALS is either a single signal or a list of signals.  Signals may be specified numerically or symbolically as per signal(3).  The SIG prefix may be omitted.  ONEXIT (signal 0) is raised upon exit from Expect.

     With no arguments, trap prints the commands associated with each signal number.

     For example, trap {send_user "Ouch!"} SIGINT prints Ouch! each time the user presses C-c.  The default behavior is restored by trap SIGINT.

     Note that output may be lost if signals arrive during reads (although this is usually the desired behavior).

     trap will not let you override the action for SIGALRM as this is used internally to Expect.  The disconnect command sets SIGALRM to SIG_IGN (ignore).  You can reenable this as long as you disable it during subsequent spawn commands.

     Few checks on signals are made.  For example, trap does not prevent you from registering signals that the kernel refuses to catch. See signal(3) for more information.

wait [-i spawn_id]
     Delays until a signal is received or the named spawned process (or the current process if none is named) terminates (or stops due to tracing). (See wait(2) for more information.)

     wait returns two integers.  The first integer is the process id of the process that was waited upon.  In this case, the second integer is WEXITSTATUS (see wait(2)).  If your system does not support WEXITSTATUS, the raw exit value is returned.  If an error occurs during execution of the wait, the integers returned are -1 followed by errno.

     The -i option declares the process to wait corresponding to the named spawn id (NOT the process id).

Pretty-Printing
===============

A vgrind definition is available for pretty-printing Expect scripts.  Assuming the vgrind definition supplied with the Expect distribution is correctly installed, you can use it as:

     vgrind -lexpect file

Examples
========

It many not be apparent how to put everything together that is described here.  I encourage you to read and try out the many examples in the example directory of the Expect distribution.  Some of them are real programs.  Others are simply illustrative of certain techniques, and of course, a couple are just quick hacks.  The INSTALL file has a quick overview of these programs.

The Expect papers (see Expect Bibliography) are also useful although invariably shorter.  However, there is a significant amount of explanatory text accompanying those examples.

Caveats
=======

Expect takes a rather liberal view of scoping.  In particular, variables read by commands specific to the Expect program will be sought first from the local scope, and if not found, in the global scope.  For example, this obviates the need to place global timeout in every procedure you write that uses expect.  On the other hand, variables written are always in the local scope (unless a global command has been issued).

If you cannot enable the multispawning capability (i.e., your system supports neither select (BSD), poll (SVr>2), nor something equivalent), Expect will only be able to control a single process at a time.  In this case, do not attempt to set spawn_id, nor should you execute processes via exec while a spawned process is running.  Furthermore, you will not be able to expect from multiple processes (including the user as one) at the same time.

Terminal parameters can have a big effect on scripts.  For example, if a script is written to look for echoing, it will misbehave if echoing is turned off.  For this reason, Expect forces sane terminal parameters by default.  Unfortunately, this can make things unpleasant for other programs.  As an example, the Emacs shell wants to change the ``usual'' mappings: newlines get mapped to newlines instead of carriage-return newlines, and echoing is disabled.  This allows one to use emacs to edit the input line.  Unfortunately, Expect cannot possibly guess this.

You can request that Expect not override its default setting of terminal parameters, but you must then be very careful when writing scripts for such environments.  In the case of Emacs, avoid depending upon things like echoing and end-of-line mappings.

The commands that accepted arguments braced into a single list (the expect variants and interact) use a heuristic to decide if the list is actually one argument or many.  The heuristic can fail only in the case when the list actually does represent a single argument which has multiple embedded \n characters with non-whitespace characters between them.  This seems sufficiently improbable; however, the argument -brace can be used to force a single argument to be handled as a single argument.  This could conceivably be used with machine-generated Expect code.

Bugs
====

It was really tempting to name the program sex (for either ``Smart EXec'' or ``Send-EXpect''), but good sense (or perhaps just Puritanism) prevailed.

Tcl 6.0 through 6.3 have a bug which may produce the error Tcl_WaitPids got unknown process followed by a core dump.  The problem is that Tcl assumes it knows about all forked processes.  When it waits for one of its own (i.e., in exec) and stumbles across one that was spawned by Expect, it generates that error.  Until this is fixed, make sure you do a wait on any spawned processes that have exited before you call exec.  system is safe from this bug, so if you do not need the differences provided by exec, you can use system meanwhile.  Tcl 6.2 has a partial fix; core is not dumped, but Expect loses the possibility of waiting on the process if your system does not support waitpid.

Since Tcl uses C-style null-terminated strings, there is no way to represent strings with nulls in them.  Expect will record such output to the log and stdout, but it will strip them out before performing string matching or storing in the variable expect_out.

When a shell is spawned on an HP-UX system, it complains about not being able to access the tty.  However, it runs anyway.  You'll have to discard that message in your scripts, though.  If you figure out why this occurs please let me know.

Ultrix 4.1 (at least the latest versions around here) considers timeouts of above 1000000 to be equivalent to 0.

Telnet (verified only under SunOS 4.1.2) hangs if TERM is not set.  This is a problem under cron and at, which do not define TERM.  Thus, you must set it explicitly---to what type is usually irrelevant.  It just has to be set to something!  The following probably suffices for most cases.

     set env(TERM) vt100

Some implementations of ptys are designed so that the kernel throws away any unread output after 10 to 15 seconds (actual number is implementation-dependent) after the process has closed the file descriptor.  Thus, Expect programs such as

     spawn date
     exec sleep 20
     expect

will fail.  To avoid this, invoke non-interactive programs with exec rather than spawn.  While such situations are conceivable, in practice I have never encountered a situation in which the final output of a truly interactive program would be lost due to this behavior.

On the other hand, Cray UNICOS ptys throw away any unread output immediately after the process has closed the file descriptor.  I have reported this to Cray and they are working on a fix.

Sometimes a delay is required between a prompt and a response, such as when a tty interface is changing UART settings or matching baud rates by looking for start/stop bits.  Usually, all this requires is to sleep for a second or two.  A more robust technique is to retry until the hardware is ready to receive input.  The following example uses both strategies:

     send "speed 9600\r";
     exec sleep 1
     expect {
       timeout {send "\r"; continue -expect}
       $prompt
     }

Expect Hints
============

There are a couple of things about Expect that may be non-intuitive. This section attempts to address some of these things with a couple of suggestions.

A common expect problem is how to recognize shell prompts.  Since these are customized differently by differently people and different shells, portably automating rlogin can be difficult without knowing the prompt. A reasonable convention is to have users store a regular expression describing their prompt (in particular, the end of it) in the environment variable EXPECT_PROMPT.  Code like the following can be used.  If EXPECT_PROMPT does not exist, this code still has a good chance of functioning correctly.

     set prompt "(%|#|\$) $"          ;# default prompt
     if [info exists env(EXPECT_PROMPT)] {
       set prompt $env(EXPECT_PROMPT)
     }

     expect -re $prompt

I encourage you to write expect patterns that include the end of whatever you expect to see.  This avoids the possibility of answering a question before seeing the entire thing.  In addition, while you may well be able to answer questions before seeing them entirely, if you answer early, your answer may appear echoed back in the middle of the question.  In other words, the resulting dialogue will be correct but look scrambled.

Most prompts include a space character at the end.  For example, the prompt from ftp is f, t, p, > and .  To match this prompt, you must account for each of these characters.  It is a common mistake not to include the blank.  Put the blank in explicitly.

If you use a pattern of the form X*, the * will match all the output received from the end of X to the last thing received.  This sounds intuitive but can be somewhat confusing because the phrase ``last thing received'' can vary depending upon the speed of the computer and the processing of I/O both by the kernel and the device driver.

In particular, humans tend to see program output arriving in huge chunks (atomically) when in reality most programs produce output one line at a time.  Assuming this is the case, the * in the pattern of the previous paragraph may only match the end of the current line, even though there seems to be more, because at the time of the match that was all the output that had been received.

expect has no way of knowing that further output is coming unless your pattern specifically accounts for it.

Even depending on line-oriented buffering is unwise.  Not only do programs rarely make promises about the type of buffering they do, but system indigestion can break output lines up so that lines break at seemingly random places.  Thus, if you can express the last few characters of a prompt when writing patterns, it is wise to do so.

If you are waiting for a pattern in the last output of a program and the program emits something else instead, you will not be able to detect that with the timeout keyword.  The reason is that expect will not time out---instead it will get an eof indication.  Use that instead.  Even better, use both.  That way if that line is ever moved around, you will not have to edit the line itself.

Newlines are usually converted to carriage return, linefeed sequences when output by the terminal driver.  Thus, if you want a pattern that explicitly matches the two lines, from, say, printf("foo\nbar"), you should use the pattern foo\r\nbar.

A similar translation occurs when reading from the user, via expect_user.  In this case, when you press return, it will be translated to a newline.  If Expect then passes that to a program which sets its terminal to raw mode (like telnet), there is going to be a problem, as the program expects a true return.  (Some programs are actually forgiving in that they will automatically translate newlines to returns, but most do not.)  Unfortunately, there is no way to find out that a program put its terminal into raw mode.

Rather than manually replacing newlines with returns, the solution is to use the command system stty raw, which will stop the translation. Note, however, that this means that you will no longer get the cooked line-editing features.

interact implicitly sets your terminal to raw mode so this problem will not arise then.

It is often useful to store passwords (or other private information) in Expect scripts.  This is not recommended since anything that is stored on a computer is susceptible to being accessed by anyone.  Thus, interactively prompting for passwords from a script is a smarter idea than embedding them literally.  Nonetheless, sometimes such embedding is the only possibility.

Unfortunately, the UNIX file system has no direct way of creating files which are executable but unreadable.  Systems which support setgid shell scripts may indirectly simulate this as follows:

Create the Expect script (that contains the secret data) as usual.  Make its permissions be 750 (-rwxr-x---) and owned by a trusted group, i.e., a group which is allowed to read it.  If necessary, create a new group for this purpose.  Next, create a /bin/sh script with permissions 2751 (-rwxr-s--x) owned by the same group as before.

The result is a script which may be executed (and read) by anyone.  When invoked, it runs the Expect script.

Expect Bibliography
===================

See Tcl, for discussion of the Tcl language that underlies Expect.

See libexpect(3), for discussion of a library of C functions implementing expect functionality.

A number of papers provide further reading:

expect: Curing Those Uncontrollable Fits of Interactivity
     Don Libes, Proceedings of the Summer 1990 USENIX Conference, Anaheim, California, June 11-15, 1990.

Using expect to Automate System Administration Tasks
     Don Libes, Proceedings of the 1990 USENIX Large Installation Systems Administration Conference, Colorado Springs, Colorado, October 17-19, 1990.

Tcl: An Embeddable Command Language
     John Ousterhout, Proceedings of the Winter 1990 USENIX Conference, Washington, D.C., January 22-26, 1990.

expect: Scripts for Controlling Interactive Programs
     Don Libes, Computing Systems, Vol. 4, No. 2, University of California Press Journals, November 1991.

Regression Testing and Conformance Testing Interactive Programs
     Don Libes, Proceedings of the Summer 1992 USENIX Conference, pp. 135-144, San Antonio, TX, June 12-15, 1992.

Kibitz---Connecting Multiple Interactive Programs Together
     Don Libes, Software---Practice & Experience, John Wiley & Sons, West Sussex, England, to appear.

Acknowledgements
================

Don Libes, of the National Institute of Standards and Technology (NIST), implemented Expect.

Thanks to John Ousterhout for Tcl, and Scott Paisley for inspiration. Thanks to Rob Savoye for Expect's autoconfiguration code.

The HISTORY file documents much of the evolution of Expect.  It makes interesting reading and might give you further insight to this software.  Thanks to the people mentioned in it who sent me bug fixes or gave other assistance.

Design and implementation of Expect was paid for by the U.S. government and is therefore in the public domain.  However the author and NIST would like credit if this program and documentation or portions of them are used.